Draft: portable GGML GPU offload for Vulkan, Metal, and CUDA - #5

Status

This is an integration-preview draft, not a request to merge the entire series
as one change. It publishes the complete tested implementation for users and
invites guidance on the proposed review split below.

The branch keeps CPU execution available and adds exact GGML device selection,
integrated-GPU acceptance, direct backend weight loading, and scheduled graph
execution for Vulkan, Metal, and CUDA. It covers full and cached masked
diffusion as well as autoregressive prefill, decode, batch, profiling, and
speculative decoding.

Commit structure

The history is intentionally linear and each commit builds and passes its
applicable model-free tests:

1. repair the existing C++ logit-validation path;
2. rank Dream remasking candidates by normalized probability;
3. preserve the predecessor rows required by Dream's shifted logits;
4. use tokenizer metadata when removing unresolved masks;
5. expose exact GGML devices, including integrated GPUs;
6. load weights and execute full diffusion on the selected backend;
7. offload cached diffusion and its K/V transfers;
8. offload autoregressive and speculative execution;
9. add backend-divergence diagnostics; and
10. expose device selection through the Python tools.

The four pre-existing bug fixes are kept free of GPU behavior. The first is
also proposed independently in #4. Standalone branches for the remaining fixes
are available in the fork but have not been opened as additional upstream PRs:

• fix/dream-probability-ranking
• fix/dream-shifted-cache
• fix/tokenizer-mask-id

Placement and backend coverage

The implementation uses GGML's device, buffer, and scheduler abstractions; it
contains no product-specific Strix Halo, Apple, or NVIDIA path. The CLI accepts
an exact device name and reports both model allocation and per-backend graph
node counts. A selected non-CPU device fails visibly if the scheduler places no
operations on it.

The cleaned tip has been freshly built and tested on:

The cleaned tip was validated through CUDA0 using CUDA 13.3 Update 1. The
exact prior history remains available as
archive/portable-gpu-offload-validated-3774fc6.

Validation

• every commit in the ten-commit series was configured, built, and tested in
  order with GCC 15.2 and GGML_NATIVE=OFF;
• the final tree passes all six CTests in the ordinary and strict
  -Wall -Wextra -Wpedantic -Werror CPU builds;
• generated F32 and Q4_K_M two-layer models pass full diffusion, full/sparse
  cache, autoregressive prefill/decode/batch/profiling, high-level generation,
  and speculative decoding on Vulkan0, MTL0, and CUDA0; quantized AR
  requires exact greedy-token equality while allowing 0.003 NMSE for
  backend-specific quantized reductions;
• the official Dream-v0-Instruct-7B Q4_K_M model loads all 339 tensors into the
  selected backend and executes full and cached production graphs with no
  observed CPU nodes on all three freshly tested GPUs;
• the production full-cache K/V NMSE is 0.000072243 on Vulkan and
  0.000063916 on Metal, and 0.000078607 on CUDA; and
• Python wrapper, malformed-output, tokenizer-mask, and device-propagation
  regressions are part of CTest.

The previously recorded matched cache-on throughput was:

CUDA no-cache reached 155.36 tok/s. These are runtime comparisons on different
hosts, not controlled GPU rankings, and were not rerun merely to reconstruct
the commit history.

Known numerical limitation

The production sparse active-set graph is shape- and backend-sensitive. In the
four-row diagnostic, logit NMSE is 0.0425762 on Vulkan, 0.0188916 on
Metal, and 0.0505584 on CUDA even though K/V remains close.
Top-token mismatches also vary by backend and shape.

The production Dream AR diagnostic places all 872 graph nodes on CUDA, but
its greedy sequence does not exactly match CPU. The generated two-layer Q4_K_M
AR test does match CPU exactly. Production AR placement is therefore evidence
of execution coverage, not a language-output parity claim.

A decoded arithmetic control returned 345 on CPU, Metal cache/no-cache, and
Vulkan no-cache, but default Vulkan sparse cache returned 355. This draft
therefore does not claim language-output parity. --no-cache is the current
correctness-sensitive control until a prompt corpus measures the approximation.
On the tested CUDA workload it was also faster than sparse caching.

Proposed review split

If there is maintainer interest, I would turn the GPU portion into three
stacked review units rather than request review of this complete integration
diff:

1. device discovery, allocation, and full masked-diffusion offload;
2. cached masked-diffusion offload; and
3. autoregressive/speculative offload plus Python device plumbing.

The focused correctness fixes can continue independently of that series.

AI assistance disclosure

I used OpenAI Codex to help investigate the defects, implement the changes,
construct the tests, and organize the commit series. I reviewed the resulting
diffs and validation output and take responsibility for the contribution.